Standalone force-catch-up for Marten under Wolverine-managed distribution (GH-3697) - #3735
Merged
Merged
Conversation
…ibution (GH-3697) Under `UseWolverineManagedEventSubscriptionDistribution` the Marten store runs in `DaemonMode.ExternallyManaged`. Since marten#4904, `IHost.ForceAllMartenDaemonActivityToCatchUpAsync()` is deliberately a passive read-only wait in that mode -- it no longer builds projections on demand, and it explicitly punts active catch-up to "the external coordinator", i.e. Wolverine. Wolverine already implements that path, in `catchUpThroughCoordinatorAsync` (GH-3349), but only exposes it as a `TrackActivity()` stage (`PauseThenCatchUpOnMartenDaemonActivity`). A suite that isn't driving its work through a tracked session had nothing to call, so it reimplemented the quiesce-then-catch-up dance by hand -- and the obvious hand-rolled version races. Adds the standalone entry point on `IHost` and `IServiceProvider`, plus the `<T>` ancillary-store variants: await host.PauseThenCatchUpOnMartenDaemonActivityAsync(); await host.PauseThenCatchUpOnMartenDaemonActivityAsync(CatchUpMode.AndDoNothing); await host.PauseThenCatchUpOnMartenDaemonActivityAsync<ILetterStore>(); Same algorithm the tracked-session stage runs, minus the message-tracking bookkeeping. The important part is what it does NOT do: it never calls `IProjectionDaemon.CatchUpAsync`. Calling that while the coordinator owns the shards is what adds a *second* writer of each progression row and produces the `ProgressionProgressOutOfOrderException` and `23505 pk_mt_event_progression` duplicate-key errors reported on the hand-rolled workaround. Resuming the agents that already own those shards and waiting for non-stale data means there is only ever one writer, so neither race exists to be retried around and swallowed. `CatchUpMode` governs the state afterwards, in a `finally`, so a timeout still restores it. Tests cover the main store, an ancillary store, and that `AndDoNothing` leaves the daemons paused and a second pass still catches up events appended while they were paused -- the exact state the hand-rolled version was fighting. Docs: a "Forcing projection catch-up outside a tracked session" section in the testing guide, including a warning against the racing pattern. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merged
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3697.
We already had most of this
Under
UseWolverineManagedEventSubscriptionDistributionthe Marten store runs inDaemonMode.ExternallyManaged, and since marten#4904IHost.ForceAllMartenDaemonActivityToCatchUpAsync()is deliberately a passive read-only wait in that mode — it punts active catch-up to "the external coordinator", i.e. Wolverine.Wolverine already implements that coordinator path —
catchUpThroughCoordinatorAsync, added in GH-3349 — but only exposes it as aTrackActivity()stage,PauseThenCatchUpOnMartenDaemonActivity(). A suite that isn't driving its work through a tracked session had nothing to call, which is exactly the gap reported.What's added
The standalone entry point, on
IHostandIServiceProvider, plus the<T>ancillary-store variants:Against the three asks in the issue:
WaitForNonStaleProjectionDataAsynccovers every database in the store.CatchUpMode, applied in afinallyso a timeout still restores it.On the two exceptions in the report
ProgressionProgressOutOfOrderExceptionand23505 ... pk_mt_event_progressionare not races the supported path swallows — they are the signature of the workaround'sdaemon.StopAllAsync()+daemon.CatchUpAsync()under a paused coordinator.CatchUpAsyncis the second writer. This helper never calls it, so neither error has anything to arise from. Nothing needs to be retried or swallowed.Tests
MartenTests/TestHelpers/standalone_catch_up_under_wolverine_distribution.cs, all green locally:UseWolverineManagedEventSubscriptionDistribution, no tracked session;AndDoNothingleaves the daemons paused, and a second pass still catches up events appended while they were paused — the exact state the hand-rolled version was fighting.Full
wolverine.slnxRelease build clean.Docs
New "Forcing projection catch-up outside a tracked session" section in the testing guide, with an explicit warning against the pause-then-
CatchUpAsyncpattern and why it produces those two errors.🤖 Generated with Claude Code